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Abstract 


We  have  been  studying  DoD  acquisition  decision  making  since  2009.  The  US  DoD 
acquisition  process  is  extremely  complex.  There  are  three  key  processes  that  must 
work  in  concert  to  deliver  capabilities:  determining  warfighters’  requirements  and 
needs;  planning  the  DoD  budget,  and  procuring  final  products.  Each  process 
produces  large  amounts  of  information  (Big  Data).  There  is  a  critical  need 
for  automation,  validation,  and  discovery  to  help  acquisition  professionals,  decision 
makers  and  researchers  understand  the  important  content  within  large  data  sets  and 
optimize  DoD  resources  throughout  the  processes.  Lexical  Link  Analysis  (LLA)  can 
help  by  applying  automation  to  reveal  and  depict — to  decision-makers — the 
correlations,  associations,  and  program  gaps  across  all,  or  subsets  of,  acquisition 
programs  examined  over  many  years.  This  enables  strategic  understanding  of  data 
gaps  and  potential  trends,  and  can  inform  managers  where  areas  might  be  exposed 
to  higher  program  risk,  and  how  resource  and  big  data  management  might  affect  the 
desired  return  on  investment  (ROI)  among  projects.  In  this  report,  we  describe  new 
developments  in  analytics  and  visualization,  how  LLA  is  adaptive  to  Big  Data 
Architecture  and  Analytics  (BDAA),  and  reveal  needs  for  Big  Acquisition  Data  used  in 
Defense  Acquisition  Visibility  Environment  (DAVE). 

Keywords:  Lexical  Link  Analysis,  big  data,  big  acquisition  data,  big  data  architecture, 
big  data  analytics 
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Executive  Summary 


We  have  been  studying  DoD  acquisition  decision  making  (Gallup,  MacKinnon,  Zhao, 
Robey  &  Odel,  2009;  Zhao,  Gallup  &  MacKinnon,  201 0,  201 1 , 2012a,  201 2b,  201 3  & 
2014)  since  2009.  The  US  DoD  acquisition  process  is  extremely  complex.  There  are 
three  key  processes  that  must  work  in  concert  to  deliver  capabilities:  definition  of 
warfighters’  requirements  and  needs;  DoD  budget  planning,  and  procurement  of 
products,  as  in  Figure  1 .  Each  process  produces  increasingly  large  volumes  of 
information  (Big  Data).  The  need  for  automation,  validation,  and  discovery  is  now  a 
critical  need,  as  acquisition  professionals,  decision  makers  and  researchers  grapple 
to  understand  data  and  make  decisions  to  optimize  DoD  resources. 


Warfighter's 

■ 

Requirementsl 

Why  and  what  to  buy: 
mission  success  at 
lowest  cost,  service 
advocacy 


DoD  Budget  Processes2 

How  much  and  when  to 
buy:  control,  oversight, 
balance 


Acquisition 
Final  Products  for 
Procurements 

Howto  buy:  balancing 
cost,  schedule, 
performance 


LLA  automates  the  possibility  to  develop  awareness  of  the 
"fit"  between  the  budget,  final  products,  and  requirements. 

1e.g.  Universal  JointTasks  Lists,  Urgent  Needs  Statements.  2e.g.  Program  Elements,  3e.g.  Weapon  Books 


Figure  1. DoD  Acquisition  Decision  Making 

We  have  been  working  on  the  problem  of  how  the  interlocking  systems  processes 
become  aware  of  their  fit  between  DoD  programs  and  warfighters’  needs  and  how 
can  gaps  of  non-fit  be  revealed?  Moreover,  in  the  performance  of  DoD  acquisition 
processes,  each  functional  community  is  required  to  review  only  the  particular 
information  for  which  they  are  responsible,  further  exacerbating  the  problem  of  lack  of 
fitness.  For  example,  the  systems  engineering  community  typically  only  examines  the 
engineering  documents  and  feasibility  studies,  the  test  and  evaluation  community 
looks  only  at  the  test  and  evaluation  plans,  and  the  acquisition  community  looks  only 
at  acquisition  strategies.  Rarely  do  these  stakeholders  review  each  other’s  data  or 
jointly  discuss  the  core  questions  and  integrated  processes  together  as  depicted  in 
Figure  1 . 
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In  this  report,  we  describe  new  developments  in  analytics  and  visualization,  how  LLA 
is  adaptive  to  Big  Data  Architecture  and  Analytics  (BDAA),  and  reveal  needs  for  Big 
Acquisition  Data  used  in  Defense  Acquisition  Visibility  Environment  (DAVE).  This 
enables:  1)  discovery  topics  and  themes;  2)  comparison,  sorting  and  ranking  data  the 
importance  of  the  data  source;  3)  strategic  understanding  of  the  correlations  of 
associations  among  data  attributes,  data  vocabularies  and  data  entities;  4)  discovery 
of  potential  trends,  patterns,  anomalies  and  data  gaps;  and  5)  correlation  of  the 
findings  with  areas  that  might  have  higher  program  risk,  and  how  resources  might  be 
managed  for  the  desired  Return  On  Investment  (ROI). 
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Background 


Motivated  by  the  lack  of  fit  and  horizontal  integration  in  the  DoD  acquisition  process, 
we  have  been  applying  Lexical  Link  Analysis  (LLA),  a  data-driven  automation 
technology  and  methodology  across  DoD  acquisition  processes  to: 

•  Surface  themes  and  their  relationships  across  multiple  data  sources. 

•  Discover  high  value  areas  for  investment. 

•  Compare  and  correlate  data  from  multiple  data  sources. 

•  Sort  and  rank  important  and  interesting  information. 

LLA  is  a  data-driven  method  for  pattern  recognition,  anomaly  detection  and  data 
fusion.  It  shares  indexes  not  data,  and  is  therefore  feasible  for  parallel  and 
distributed  processing,  adaptive  to  Big  Data  Architecture  and  Analytics  (BDAA)  and 
capable  of  analyzing  Big  Acquisition  Data. 

As  a  motivating  example  from  past  work,  we  conducted  a  detailed  examination  of 
the  Research,  Development,  Test  and  Evaluation  (RDT&E)  budget  modification 
practice  from  one  year  to  the  next,  over  the  course  of  ten  years  and  450  DoD 
Program  Elements.  We  found  a  pattern  revealing  that  the  programs  with  fewer  links 
(measured  by  LLA)  to  warfighters’  requirements,  received  more  budget  reduction  in 
total  but  less  on  average,  indicating  that  budget  reduction  may  have  focused  mainly 
on  large  and  expensive  programs  rather  than  perhaps  cutting  all  the  programs  that 
do  not  match  warfighters’  requirements.  Furthermore,  the  programs  with  more  links 
to  each  other  received  more  budget  reduction  in  total,  as  well  as  on  average, 
indicating  a  pattern  of  good  practice  of  allocating  DoD  acquisition  resources  to  avoid 
overlapping  efforts  and  to  fund  new  and  unique  projects.  These  findings  were  useful 
as  validation  and  guidance  for  future  decision  processes  for  automatically  identifying 
programs  to  match  warfighter’s  requirements,  limit  overall  spending,  maximize 
efficiencies,  eliminate  unnecessary  costs  and  maximize  the  return  on  investment. 
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Methodology 

Lexical  Link  Analysis  (LLA) 

LLA  has  been  used  to  analyze  unstructured  and  structured  data  for  pattern 
recognition,  anomaly  detection,  and  data  fusion.  It  uses  the  theory  of  System  Self- 
Awareness  (SSA)  to  identify  high-value  information  in  the  data  that  can  be  used  to 
guide  future  decision  processes  in  a  data-driven  or  unsupervised  learning  fashion.  It 
is  implemented  via  a  smart  infrastructure  named  “system  and  method  for  knowledge 
pattern  search  from  networked  agents  (US  patent  8,903,756)”  also  known  as 
Collaborative  Learning  Agents  (CLA),  licensed  from  Quantum  Intelligence,  Inc. 

(Zhou,  Zhao,  and  Kotak,  2009). 

In  LLA,  a  complex  system  is  expressed  in  specific  vocabularies  or  lexicons  to 
characterize  its  features,  attributes  or  its  surrounding  environment.  LLA  uses  bi¬ 
gram  word  pairs  as  the  features  to  form  word  networks.  Figure  2  depicts  using  LLA 
to  analyze  ten  years  of  the  reports  in  the  NPS  Acquisition  Program  with  word  pairs 
as  groups  or  themes.  Figure  3  shows  a  detail  of  a  theme  in  Figure  2.  A  node 
represents  a  word.  A  link  or  edge  represents  a  word  pair. 


Figure  2.  Themes  Discovered  in  Colored  Groups 
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Figure  3.  A  Detailed  View  of  a  Theme  in  Figure  2 

LLA  is  related  to  bags-of-words  (BAG)  methods  such  as  LDA  (Blei,  Ng  &  Jordan, 
2003)  and  text-as-network  (TAN)  methods  such  as  the  Stanford  Lexical  Parser 
(SLP;  SNLPG,  2015).  LLA  selects  and  groups  features  into  three  basic  types: 

•  Popular  (P):  Main  themes  found  from  the  data.  Figure  3  is  an  example  of  a 
popular  theme  centered  around  word  nodes  “analysis,  model,  and  approach.” 
These  themes  could  be  less  interesting  because  they  are  already  in  the  public 
consensus  and  awareness.  They  represent  the  patterns  in  the  data. 

•  Emerging  (E):  Themes  that  may  grow  to  be  popular  over  time.  Figure  4  is  an 
example  of  an  emerging  theme  centered  around  word  nodes  “national, 
defense,  and  acquisition.” 

•  Anomalous  (A):  Themes  that  may  be  off-topic  but  that  may  be  interesting  for 
further  investigation.  Figure  5  is  an  example  of  anomalous  theme  centered 
around  word  nodes  “stock,  and  market(s).” 
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Figure  5.  An  Example  of  Anomalous  Theme 

Figure  6  summarizes  LLA  used  for  historical  and  new  data.  The  red  part  shows  a 
pattern  (e.g.,  a  theme)  discovery  phase  using  historical  data  including  data  fusion 
that  come  from  multiple  learning  agents.  The  black  part  shows  an  application  phase 
that  a  new  data  is  compared  with  the  patterns  discovered  and  hence  the  anomalies 
are  revealed. 
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Figure  6.  Diagram  for  the  LLA  Method 

Word  Pairs  Generalization  and  CCC  Method 

Figure  7  shows  the  word  pairs/bi-gram  in  an  LLA  can  be  generalized  as  a  Context- 
Concept-Cluster  (CCC)  model,  where  a  context  is  a  generic  attribute  which  can  be 
shared  by  multiple  data  sources,  a  concept  is  a  specific  attribute  for  a  data  source 
and  a  cluster  is  a  combination  of  attributes  or  themes  that  can  be  computed  using  a 
word  community  finding  algorithm  (e.g.,  Girvan  &  Newman,  2002)  in  Figure  6  to 
characterize  a  data  set.  Context  can  be  a  word,  location,  time  or  object,  etc. 


Figure  7.  The  Word  Pairs/Bi-gram  in  LLA  a  Context-Concept-Cluster  (CCC)  Model 
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Figure  8  summarizes  how  a  generalized  CCC  method  is  used  for  historical  and  new 
data.  Similar  to  Figure  6  there  is  a  pattern  discovery  phase  using  historical  data 
where  patterns  are  learned  and  discovered,  and  an  application  phase  for  a  new  data 
is  compared  with  the  patterns  discovered  and  anomalies  are  revealed. 


Figure  8:  Diagram  for  the  CCC  Method 
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Research  Results 


In  this  section,  we  describe  the  research  results  for  the  following  two  tasks 
completed  this  year. 

Task  1.  We  worked  with  the  OSD  OUSD  ATL  (US)  to  install  the  LLA/SSA/CLA 
system  as  a  web  service  in  the  Defense  Acquisition  Visibility  Environment  (DAVE) 
test  bed.  It  was  approved,  installed  and  in  progress  for  source  code  security 
scanning.  The  goal  has  been  to  bring  the  tool  close  to  the  DAVE  sources  as  shown 
in  Figure  9. 


•  DAVE  interfaces  with  authoritative  data  sources  to  support  the  OUSD(AT&L) 

and  his  staff  in  decision-making,  oversight  and  analysis  for  major  programs 

•  Sample  data  sources 

-  Defense  Technical  Information 

-  Acquisition  Information  Repository 

Center  (DTIC) 

(AIR) 

o  Program  Elements  (PEs): 

O 

Test  &  Evaluation  Master  Plan  (TEMP) 

[http://www.dtic.miI/descriptivesum/l 

O 

Systems  Engineering  Plan  (SEP) 

-  Defense  Acquisition  Management 

O 

Life  Cycle  Sustainment  Plan  (LCSP) 

Information  Retrieval  (DAMIR) 

(http://www.acq.osd.mil/damir/) 

O 

Acquisition  Strategy  Reports  (ASR) 

o  Selected  Acquisition  Report  (SAR) 

O 

Acquisition  Decision  Memorandum 
(ADM,  for  Milestone  B  2366b 

o  Defense  Acquisition  Executive 

Certification) 

Summary  (DAES) 

O 

Technology  Readiness  Assessment 

o  Acquisition  Program  Baseline  (APB) 

(TRA),  and  LCSP 

Figure  9.  Defense  Acquisition  Visibility  Environment  (DAVE) 


Figure  10  shows  the  existing  LLA  reports  and  visualizations  in  the  LLA  installed.  The 
details  of  the  existing  visualizations  can  be  found  in  the  earlier  publications  (Zhao, 
Gallup  &  MacKinnon,  201 3,  2014  &  201 5).  Figure  11,  12,  13,  and  1 4  are  the  new 
visualizations  developed  this  year  using  the  Data-Driven  Documents  (D3)  tool. 

Figure  1 1  shows  multiple  bar  and  pie  charts  for  multiple  themes  discovered  using 
LLA  when  comparing  two  data  sources,  Specifically,  it  reveals:  how  many  lexical 
terms  are  matched  (red)  and  how  many  lexical  terms  are  unique  for  each  data 
source  (green)  and  in  each  theme.  Figure  12  shows  the  lexical  terms  in  each  theme 
in  a  collapsible  tree.  Figure  1 3  shows  the  lexical  terms  in  a  word  cloud  with  each 
color  representing  a  theme.  Figure  14  shows  clusters  of  data  sources  in  a 
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hierarchical  edge  bundling.  The  clusters  are  generated  based  on  the  numbers  of 
matched  and  unique  lexical  terms  found  among  the  data  sources. 


Figure  10.  Example  LLA  Reports  and  Visualizations 


Figure  11.  D3  Visualization  Bar  and  Pie  Charts:  The  Number  Lexical  Terms  Matched  in 
Red  and  Number  of  Lexical  Terms  Unique  in  Green 
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Figure  12.  D3  Visualization  -  Collapsible  Tree:  Lexical  Terms  in  Each  Theme  Listed  in 

an  Interactive  and  Collapsible  Tree 
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Figure  13.  D3  Visualization  -  Word  Cloud:  Each  Color  Representing  a  Theme, 
Filters  and  Interactions  on  Link  Weight,  Scale  and  Attraction 
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Figure  14.  D3  Visualization  -  Hierarchical  Edge  Bundling:  Showing  Clusters  of  Data 
Sources  (e.g.,  Index  Technology,  etc.).  The  Clusters  and  Links  Among  Data  Sources 

Measured  by  the  Number  of  Lexical  Terms 


Task  2.  We  also  explored  how  to  use  LLA  jointly  with  other  business  intelligence 
tools  especially  Big  Data  Architecture  and  Analytics  (BDAA)  tools: 

Transfer  learning,  unsupervised  learning  and  deep  learning  across  heterogeneous 
data  sources  are  trends  for  the  academic  and  commercial  Big  Data  applications. 
Deep  Learning  includes  unsupervised  machine  learning  techniques  (e.g.,  neural 
networks)  for  recognizing  objects  of  interest  from  Big  Data,  for  instance,  sparse 
coding  (Olshausen  &  Field,  1996)  and  self-taught  learning  (Raina,  Battle,  Lee, 
Packer, &  Ng,2007).  The  self-taught  learning  approximates  the  input  for  unlabeled 
objects  as  a  succinct,  higher-level  feature  representation  of  sparse  linear 
combination  of  the  bases.  It  uses  the  Expectation  and  Maximization  (EM)  method  to 
iteratively  learn  coefficients  and  bases.  Deep  Learning  links  machine  vision  and  text 
analysis  smartly.  For  example,  text  analysis  Latent  Dirichlet  Analysis  (LDA)  is  a 
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sparse  coding  where  a  bag  of  words  used  as  the  sparsely  coded  features  for  text 
(Olshausen  &  Field,  1996).  Our  methods  Lexical  Link  Analysis  (LLA),  System-Self- 
Awareness  (SSA),  and  Collaborative  Learning  Agents  (CLA)  can  be  viewed  as 
unsupervised  learning  or  Deep  Learning  for  pattern  recognition,  anomaly  detection, 
and  data  fusion. 

We  explored  a  few  deep  learning  techniques;  in  particular,  a  recursive  data  fusion 
methodology  leveraging  LLA,  SSA,  and  CLA  was  examined  as  follows: 

•  An  agent  j  represents  a  sensor,  operates  on  its  own  like  a  decentralized  data 
fusion,  however  it  does  not  communicate  with  all  other  sensors  but  only  with 
the  ones  that  are  its  peers.  A  peer  list  can  be  specified  by  the  agent. 

•  An  agent  j  includes  a  learning  engine  CLA  that  collects,  analyzes  from  its 
domain  specific  data  knowledge  base  b(t,j),  for  examples,  b(t,j)  may 
represent  the  statistics  for  bi-gram  feature  pairs  (word  pairs)  computed  from 
LLA. 

•  An  agent  j  also  includes  a  fusion  engine  SSA  with  two  algorithms  SSA1  and 
SSA2  that  can  be  customized  externally.  SSA1  integrates  the  local 
knowledge  base  b(t,j)  to  the  total  knowledge  base  B(t,j)  that  can  be  passed 
along  to  its  peers  and  used  globally  in  the  recursion  in  Figure  6.  SSA2 
assesses  the  total  value  of  the  agent  j  by  separating  the  total  knowledge  base 
into  the  categories  of  patterns,  emerging  and  anomalous  themes  based  on 
the  total  knowledge  base  B(t,j)  and  generates  a  total  value  V(t,j)  as  follows: 

-  Step  1 :  B(t,j)  =  SSA  1  (B(t- 1,  p(j)),  b(tj)); 

-  Step  2:  V(t,j)  =  SSA2(B(t,j)) 

Where  p(j)  represents  the  peer  list  of  agent  j. 

•  The  total  value  V(t,j)  is  used  in  the  global  sorting  and  ranking  of  relevant 
information. 

In  this  recursive  data  fusion,  the  knowledge  bases  and  total  values  are  completely 
data-driven  and  automatically  discovered  from  the  data.  Each  agent  has  the  exact 
same  code  of  LLA,  SSA,  and  CLA,  yet  has  its  own  data  apart  from  other  agents. 

This  agent  work  has  the  advantages  of  both  decentralized  and  distributed  data 
fusion.  It  performs  learning  and  fusion  simultaneously  and  in  parallel.  Meanwhile,  it 
categorizes  the  patterns  and  anomalous  information. 


We  have  also  met  the  acquisition  professionals  and  discussed  how  LLA  and  BDAA 
can  be  applied  to  the  DoD  acquisition  process.  The  following  list  is  a  summary  of 
business  applications  showing  how  Big  Data  analytic  tools  such  as  LLA  can  help 
decision  maker’s  understanding  of  Big  Acquisition  Data: 

•  Examine  and  compare  the  important  acquisition  data  sources  to  gain 
business  insights  to: 

-  Study  prime  and  subcontractor  relationships. 
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-  Analyze  DOT&E  Annual  Reports  using  horizontal  analysis. 

-  Compare  DOT&E  and  DAES:  What  did  the  OT  results  reveal  and  did 
issues  surface? 

•  Did  these  issues  show  up  somewhere  in  DAES  earlier? 

-  Perform  Time  series  analysis. 

•  Can  LLA  look  at  program  ratings  over  time? 

-  Compare  budget  data  and  contracts  data. 

Other  areas  of  Big  Data  exploration  may  include: 

-  Understanding  how  in  the  current  acquisition  process,  a  small  delay  or 
anomaly  in  a  contract  negotiation  process  can  have  a  huge  impact  in 
its  performance,  therefore  cost  the  government  a  lot  of  money 
downstream. 

-  Applying  BDAA  such  as  LLA  for  pattern  recognition  and  anomaly 
detection  for  these  kind  of  problems  to  enable  early  warnings  and 
predictions  to  prevent  the  downstream  risks. 

-  Leveraging  Big  Acquisition  Data  to  include  programs'  cost,  SAR, 
DAMIR,  technical  data,  to  perhaps  include  outside  economic 
environment  data  if  access  is  possible. 

-  Learning  causes  of  the  deviations  from  the  normal  behaviors  for  the 
programs/contracts  and  how  we  might  use  models  arising  from 
physics  (e.g.,  fluid  dynamics  theories). 

-  Exploring  LLA's  network  perspectives  and  social  dynamics  found 
among  the  nodes  and  the  System  Self-Awareness  (SSA)  theory  and 
how  they  may  be  used  to  lay  out  the  academic  rigor  for  business 
processes,  for  example,  to  answer  the  following  questions: 

•  Are  some  nodes  drawn  towards  some  other  nodes  because  the 
other  nodes  are  more  powerful,  for  instance,  the  powerful  nodes 
can  represent  the  organizations  with  more  decision  makers? 

•  Should  the  growth  pattern  be  attached  to  popular  existing  ideas 
(i.e.,  preferential  attachment),  or  innovative  ideas  (i.e.,  expertise 
growth)?How  are  the  forces  of  the  nodes  modeled  and  mapped 
into  the  social  network  settings  and  actual  business  processes? 
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Conclusions 


On  September  17,  2015,  we  met  with  Mr.  Mark  E.  Krzysko,  Deputy  Director, 
Acquisition,  Resources  and  Analysis  Enterprise  Information  at  the  OUSD  (AT&L).  In 
that  discussion,  we  agreed  that  future  work  will  also  include  installing  LLA  at  the 
Defense  Manpower  Defense  Center  (DMDC);  a  Federally  Funded  Research  and 
Development  Center  [ FFRDC] )  and  setting  up  a  BDAA  consortium,  whereby  DAVE 
could  provide  an  authoritative  yet  secure  data  throughput  to  the  DMDC  and  NPS 
students,  faculty,  and  acquisition  research  professionals.  This  in  turn,  can  inspire  a 
large  community  of  researchers  and  NPS  students,  who  may  be  future  leaders  of  the 
DoD  acquisition  community,  to  use  the  Big  Data  tools  and  analytics  like  LLA  to 
access  the  DAVE  data  to  answer  important  research  questions  and  advance  the 
state-of-the-art  of  DoD  acquisition. 

As  a  proof-of-concept,  we  installed  the  LLA  tool  in  the  OUSD(AT&L)  e-business 
center  to  access  multiple  categories  of  program  data  as  authentic  and  authoritative 
data  sources  in  DAVE.  We  also  enhanced  the  LLA  analytical  capabilities,  reports 
and  visualizations  to  show  it  is  a  tool  for  Big  Data  Acquisition  for  understanding  the 
insight  of  the  current  acquisition  business  processes,  answer  important  business 
questions  that  lead  to  the  identification  of  future  specific  and  productive  directions  for 
acquisition  research. 

We  are  at  the  onset  of  many  exciting  advances  in  understanding  more  about  the 
DoD  acquisition  system  by  leveraging  Big  Data  methodology  including  LLA  and  we 
look  forward  to  further  in-depth  engagement  in  the  years  to  come. 
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